1. Field of the Invention
This invention relates to the field of video stream reconstruction. In particular, the invention relates to methods and apparatuses for reconstructing low frame rate video conferencing data.
2. Description of the Related Art
The bandwidth requirements for uncompressed video streams, such as for video conferencing, can easily exceed channel bandwidth. For example, the public switched telephone network supports a maximum of approximately 56 Kbits/second. However, the common video conferencing format of quarter common intermediate format (QCIF) at 30 frames/second and 24 bits per pixel would requires 18.3 Mbits/second if uncompressed. Thus, the uncompressed video stream would require over 300 times more bandwidth than is available.
For this reason, a number of compression schemes have been developed for moving video. Generically, these compression schemes are motion compensated waveform coders. This class of coders includes several standard encoding techniques such as Moving Pictures Experts Group Level 1 (MPEG-1), MPEG-2, MPEG-4, H.261, and H.263. The basic approach of these various motion compensated coding techniques is the same, a current frame can be predicted from a previous framexe2x80x94or later framesxe2x80x94using motion information.
Thus, instead of transmitting each frame in a video stream, only a small number of frames need to be sent in full. Instead, most frames are represented by a set of motion vectors that determine for each pixel in the current frame, their location in the previous frame. Thus, for example, the first frame in a sequence might be transmitted to the receiving computer, but the second frame might be represented by vectors describing movement of blocks from previous frames.
The different standards are each targeted at various applications. For example, MPEG-1 is directed primarily to compact disc based video whereas MPEG-4 and H.263 are directed to lower bit ratexe2x80x94and frame ratexe2x80x94encoding systems suitable for video conferencing. However, even using a standard such as H.263, the low frame rate may cause the motion to appear unnatural, or jerky. As the frame rate of the drops, this problem is further exacerbated.
One solution is to have the receiving computer interpolate additional frames. For example, if the transmitted frame sequence is {1, 2, 3, 4, . . . }, the receiver can interpolate additional frames, {1.5, 2.5, 3.5, 4.5, . . . } to be shown between the transmitted frames. However, the appearance of the reconstructed stream with the additional frames can still be fairly unnatural and jerky.
Also, the quality of the reconstructed video stream is limited by the quality of the search for motion vectors performed by the transmitting computer. If for example, the transmitting computer does not have the computational power to perform an adequate search, the quality of the reconstructed frames will suffer. If an interpolation technique is used to create additional frames, the unnaturalness and jerkiness of the motion may be more apparent with fewer motion vectors.
The previous techniques do not allow for the interpolation of frames into low rate video streams in a manner that provides a natural appearance to the motion. Accordingly, what is needed is an improved method for interpolation of low frame rate video streams.
Embodiments of the invention are directed to improving the playback quality of low frame rate motion compensated waveform encoded audio/video streams. Typical examples of motion compensated waveform encoded audio/video streams include Moving Pictures Experts Group Level 1 (MPEG-1), MPEG-2, MPEG-4, H.261, and H.263 encodings. Embodiments of the invention can operate on the decoding, or receiving, side without the need for modifying the encoding, or transmitting, side. Thus, the playback quality of a standard MPEG-2 video stream can be enhanced by the system playing the video stream. These playback quality improvements are particularly useful for video conferencing audio/video streams and/or other audio/video streams that are transmitted at low frame rates, e.g. using H.263.
In order to improve playback quality, some embodiments of the invention re-encode a motion compensated waveform encoded audio/video stream at the receiving system. This allows for the detection of the movement of regions between frames that was not detected by the transmitting, encoding, side. For example, consider a region of a frame in which a person is in the process of waving her hand. The transmitting system may not have adequate processing power to complete an exhaustive search to detect the location of the hand region between frames. As such, the encoded video stream will lack motion vectors completely describing the motion of the hand.
However, at the receiving side, the computer can re-encode the stream, after decoding the stream, to perform a more exhaustive search for motion vectors. Thus, the movement of the hand region between frames can be more fully described. That additional region movement information can then be used to produce better quality playback in conjunction with motion compensated interpolation.
Some embodiments of the invention include improved motion compensated interpolation processes. These improved processes can be used in conjunction with, or separately from, the re-encoding process to improve the quality of the additional frames. The improved motion compensated interpolation processes reduce unnatural and jerky motion sometimes introduced by motion compensated interpolation. This improvement allows for the use of repeated interpolation and replacement frames to correct for inaccurate interpolated frames. This process prevents jerky motion between the interpolated frames and the actual frames.